Learning pullback HMM manifolds

Recent work in action recognition has exposed the limitations of methods which directly classify local features extracted from spatio-temporal video volumes. In opposition, encoding the actions’ dynamics via generative dynamical models has a number of desirable features when it comes to unsupervised learning of plots, crowd monitoring, and description of more complex activities. However, using fixed, all-purpose distances to classify dynamical models does not necessarily deliver good classification results. In this paper we propose a general framework for learning Riemannian metrics or distance functions for dynamical models, given a training set which can be either labeled or unlabeled. The optimal distance function is selected among a family of pullback ones, induced by a parameterized automorphism of the space of models. We focus here on hidden Markov models, study their manifold and design automorphisms there which allow us to build parametric families of metrics we can optimize upon. Experimental results are presented which show how pullback learning greatly improves action recognition performances with respect to base distances.